1. Field of the Invention
The present invention generally relates to image processing devices, and particularly relates to an image processing device which decodes moving pictures digitally compressed through such schemes as MPEG (moving picture expert group).
Broadcasting based on MPEG-digital-compression techniques such as digital-satellite broadcasting has been making market progress in recent years. The digital-compression techniques enable an efficient use of spectrum space for broadcasting, and viewers are given free choices of favorite programs from hundreds of channels. In making choices of favorite programs, viewers would prefer having a chance to take a look at aired programs simultaneously and compare them, rather than relying on printed program information such as a TV guide. To render such services to the viewers, a plurality of programs should be displayed on a TV screen in a form of multi-windows after a size reduction of each program picture. The present invention relates to a circuit which produces such a multi-window display image by drawing on functions of MPEG decoders.
2. Description of the Related Art
FIG. 1 is a block diagram of a related-art image processing device which receives moving pictures digitally compressed through MPEG and shows the moving pictures on a display. The image processing device of FIG. 1 includes an input buffer 10, a variable-length-decoding unit 11, an inverse-quantization unit 12, an inverse-DCT (discrete cosine transform) unit 13, an adder 14, a prediction unit 15, a memory controller 16, an MPEG-decoding-purpose memory 17, an output circuit 18, a display 19, an output circuit 20, a low-pass filter 21, a decimating circuit 22, a multi-window memory 23, a multi-window-control unit 24, and a bus 30. The MPEG-decoding-purpose memory 17 includes a VBV buffer 17-1, an I/P-picture buffer 17-2, a I/P-picture buffer 17-3, and a B-picture buffer 17-4. Here, MPEG-digitally-compressed-image information refers to image information which is compressed in accordance with the MPEG-international-image-compression standard. This standard employs variable-length coding for representing frequently appearing symbols by short codes and rarely appearing symbols by long codes to reduce the number of bits in a decoded code sequence. It also employs DCT for transforming an image into spatial frequency components representing vertical frequencies and horizontal frequencies of the image, quantization for quantizing the DCT coefficients by using bits in a commensurate number with image characteristics, DPCM (differential pulse code modulation) for transmitting a differential image between a current image and a previous image to reduce the amount of image information, etc.
The input buffer 10 latches the MPEG-digitally-compressed-image information. The latched image information is supplied to and temporarily stored in the VBV buffer 17-1 of the MPEG-decoding-purpose memory 17 via the bus 30 and the memory controller 16. In the MPEG standard, compressed image information has different numbers of bits allocated to different frames. Namely, some frames are represented by a relatively small number of bits, and other frames are represented by a relatively large number of bits. Since image information having different numbers of bits in different frames is transmitted at a constant bit rate (transmission rate), the input buffer 10 receives image information by spending different lengths of time for different frames. The VBV buffer 17-1 is provided for a purpose of absorbing this variation in the frame transmission time. The VBV buffer 17-1 receives the image information from the input buffer 10 when the input buffer 10 receives this image information at a constant transmission rate. The VBV buffer 17-1 then temporarily stores the image information to absorb the variation in the frame transmission time. Each frame of the image information stored in the VBV buffer 17-1 is read each time the display 19 displays one frame, and is subjected to a subsequent decoding process. International standards set forth that the number of bits stored in the VBV buffer 17-1 should be managed so as not to cause overflow or underflow of the buffer. The storage capacity of the VBV buffer 17-1 is also defined by the international standards.
Images stored in the VBV buffer 17-1 are supplied to the variable-length-decoding unit 11 via the memory controller 16 and the bus 30. The variable-length-decoding unit 11 decodes variable-length codes into fixed-length codes, and, also, extracts motion vectors, quantized DCT coefficients, etc. The inverse-quantization unit 12 receives the quantized DCT coefficients, and applies inverse-quantization to produce DCT coefficients. The inverse-DCT unit 13 receives the DCT coefficients, and applies inverse DCT to convert the image information from the frequency domain to the spatial domain.
Coding on the sender side and decoding (image reconstruction) on the receiver side are carried out for each block of an image after dividing the image into a plurality of blocks each having 16.times.16 pixels. This block is called a macro block. Motion vectors between images are also obtained for each macro block. Each macro block is further divided into smaller blocks of 8.times.8 pixels, and DCT on the sender side and inverse DCT on the receiver side are applied to each one of these smaller blocks.
MPEG-digital-image compression produces three types of coded pictures. These three types are I (intra) pictures, P (predictive) pictures, and B (bidirectionally predictive) pictures. The I pictures are obtained by applying predictive coding within a current frame. The P pictures are obtained by effecting predictive coding of a current frame by generating a differential image between the current frame and a reference frame with help of a motion vector, after obtaining the motion vector between the current frame and a previous frame serving as the reference frame. The B pictures are obtained by effecting predictive coding of a current frame by generating a differential image between the current frame and two reference frames based on motion vectors, which are detected between the current frame and the two reference frames, one in the future and the other in the past.
The past reference frame is an I picture or a a P picture in the past, and the future reference frame is a P picture in the future. When coding a B picture, a future P picture to serve as the future reference frame is coded and transmitted before this B picture is coded and transmitted. The receiver (decoder) side decodes the B picture by using a past I or P picture and the future P picture which has been received in advance. Such a decoding process is carried out based on motion vectors, and the prediction unit 15 controls the decoding process.
In the case of decoding an I picture, the adder 14 passes the image information from the inverse-DCT unit 13 without making any change to the image information. The I picture passing through the adder 14 is stored in the I/P-picture buffer 17-2 of the MPEG-decoding-purpose memory 17 via the bus 30 and the memory controller 16. The I picture stored in the I/P-picture buffer 17-2 is subsequently used for reconstructing a following P or B picture. For the purpose of display, the I picture is also supplied to the output circuit 20 via the bus 30.
In the case of decoding a P picture, a current frame (P picture) stored in the VBV buffer 17-1 is read by the variable-length-decoding unit 11, and, then, supplied to the adder 14 via the inverse-quantization unit 12 and the inverse-DCT unit 13. Each block of the current frame represents differentials from a corresponding block of a past reference frame. The past reference frame (P picture or I picture) has been already decoded, and is stored in the I/P-picture buffer 17-2 or the I/P-picture buffer 17-3. Motion vectors detected by the variable-length-decoding unit 11 are supplied to the prediction unit 15, which then fetches the past reference frame via the memory controller 16 and the bus 30. Based on the motion vectors, the prediction unit 15 provides the adder 14 with a block of the past reference frame corresponding to a given block of the current frame. The adder 14 adds each block of the current frame to a corresponding block of the past reference frame to reconstruct an image of the current frame. The reconstructed P picture is then stored in the I/P-picture buffer 17-2 or the I/P-picture buffer 17-3. The stored P picture is subsequently used for reconstructing a following P or B picture. For the purpose of display, the P picture is also supplied to the output circuit 20 via the bus 30.
In the case of decoding a B picture, a current frame (B picture) stored in the VBV buffer 17-1 is read by the variable-length-decoding unit 11, and, then, is supplied to the adder 14 via the inverse-quantization unit 12 and the inverse-DCT unit 13. Each block of the current frame represents differentials from a corresponding block of a past reference frame and a corresponding block of a future reference frame. The past reference frame (P picture or I picture) and the future reference frame (P picture or I picture) have been already decoded, and are stored in the I/P-picture buffer 17-2 or the I/P-picture buffer 17-3. Motion vectors detected by the variable-length-decoding unit 11 are supplied to the prediction unit 15, which then fetches the past reference frame and the future reference frame via the memory controller 16 and the bus 30. Based on the motion vectors, the prediction unit 15 provides the adder 14 with a block of the past reference frame and a block of the future reference frame corresponding to a given block of the current frame. The adder 14 adds each block of the current frame to corresponding blocks of the past and future reference frames to reconstruct an image of the current frame. The reconstructed B picture is then supplied to the output circuit 20 via the bus 30 for display of the image. The B-picture buffer 17-4 is used for storing an image in process of decoding.
In general, I pictures are transmitted once every 0.5 seconds (i.e., at a rate of one frame in every 15 frames). I pictures generally have the lowest compression rate and require the largest amount of information (the largest number of bits) for representing images, compared to P pictures and B pictures, although the compression rate and the information amount depend on inter-frame correlation in a time dimension. B pictures have the highest compression rate and require the smallest amount of information (the smallest number of bits) for representing images. P pictures have a compression rate higher than that of the I pictures and lower than that of B pictures.
Distinction of I/P/B pictures is made by the variable-length-decoding unit 11 extracting picture headers from a MPEG data stream and looking into the following portion of the data stream.
In FIG. 1, the low-pass filter 21, decimating circuit 22, the multi-window memory 23, and multi-window-control unit 24 are provided to implement multi-window-screen display or size-reduced-picture display. The multi-window-screen display, for example, shows pictures of a number of channels on a screen after size reductions of each picture, so that viewers can select a channel broadcasting a program matching with their tastes. The size-reduced-picture display shows broadcasted pictures of a given channel (program) on a screen after reducing the size of the pictures, using a remaining space of the screen for displaying relevant information on the program contents. Such multi-window-screen display or size-reduced-picture display requires decimation of image pixels to produce a size-reduced image. A simple decimation of image pixels, however, results in aliasing noise. This aliasing noise is created when high-frequency components of an image are turned into corresponding low-frequency components due to longer sampling intervals resulting from the decimation of pixels, and is a well-known noise in the field of the signal processing. In order to avoid the aliasing noise, high-frequency components are typically removed from the image prior to the pixel decimation.
The low-pass filter 21 of FIG. 1 removes high-frequency components of images to prevent aliasing noise. FIGS. 2A through 2C are block diagrams of examples of digital filters which are used as the low-pass filter 21. These digital filters include delay elements 31 through 36 and adders 41 through 44. Removing high-frequency components by use of such digital filters can prevent aliasing noise from generating even when pixels are decimated. The images with high-frequency components thereof being cut by the low-pass filter 21 are supplied to the decimating circuit 22. The decimating circuit 22 extracts every other pixel of the supplied images, for example, so as to generate size-reduced images having a quarter of the original size (half size in a vertical direction and half size in a horizontal direction).
The multi-window-control unit 24 feeds image-writing addresses to the multi-window memory 23 so that the images from the decimating circuit 22 are written at specified addresses. In the case of the multi-window-screen display, size-reduced images are written into the multi-window memory 23 successively at different addresses corresponding to screen positions of the size-reduced images. When reading the images, the multi-window-control unit 24 supplies reading addresses to the multi-window memory 23. The screen information stored in the multi-window memory 23 is then read in a raster scanning manner to be supplied to the output circuit 18. This screen information is shown on the display 19. The multi-window-screen display or the size-reduced-picture display is implemented in this manner.
The image processing device of FIG. 1 has a problem in that despite the presence of the MPEG-decoding-purpose memory 17, another frame memory (multi-window memory 23) becomes necessary. In the manufacturing of image processing devices, it is preferable to make all parts of an image processing device on one chip. Since the configuration of FIG. 1 requires additional frame memory, the multi-window memory 23, the multi-window-control unit 24, etc., are to be implemented on a separate chip from the chip bearing an MPEG-decoding unit, which comprises the variable-length-decoding unit 11, the inverse-quantization unit 12, the inverse-DCT unit 13, the adder 14, the prediction unit 15, the memory controller 16, and the MPEG-decoding-purpose memory 17.
Another problem of the image processing device of FIG. 1 is the need for providing the low-pass filter 21 for the removal of high-frequency components. The low-pass filter 21 is used for applying frequency-domain signal processing to a decoded image in the spatial domain, although the MPEG images were transmitted in a form of frequency components after these images had been transformed into the frequency domain. The manner in which this signal processing is carried out is thus redundant. Further, low-pass filtering in a vertical direction of the image (direction perpendicular to the scanning direction of the image) requires digital filters of a large circuit size. In the low-pass filtering in the scanning (horizontal) direction of the image, the delay elements 31 through 36 of the digital filters in FIGS. 2A through 2C can be a register with a storage capacity of only one pixel, so that the circuit size tends to be small. In the low-pass filtering in the vertical direction of the image, however, the delay elements 31 through 36 need to be a line memory with a storage capacity for storing pixels of one scan line. This requires a large circuit size of the digital filters.
Accordingly, there is a need in the size-reduced-picture display of compressed digital images for a device and a method which can generate a size-reduced image through decimation of pixels with efficient use of memories and efficient processing of image data.